COMMON INVARIANT SUBSPACE AND COMMUTING 

MATRICES 



GERALD BOURGEOIS 



Abstract. Let K be a perfect field, L be an extension field of K and A, B G 
M n (K). If A has n distinct eigenvalues in L that are explicitly known, then we 
can check if A, B are simultaneously triangularizable over L. Now we assume 
that A, B have a common invariant proper vector subspace of dimension k 
over an extension field of K and that xa> the characteristic polynomial of A, 
is irreducible over K. Let G be the Galois group of \A- We show the following 
results 

(i) If A: £ {1, n — 1}, then A, B commute. 

(ii) If 1 < k < n - 1 and G = S n or G = An, then AB = BA. 

(iii) If 1 < k < n — 1 and n is a prime number, then A_B = BA. 

Yet, when n = 4, A: = 2, we show that A, £? do not necessarily commute if 
G is not S4 or A4. Finally we apply the previous results to solving a matrix 
equation. 



1. Introduction 

Throughout this paper, K denotes a perfect field, and K an algebraic closure of 
K. Recall that a field K is said to be perfect if every irreducible polynomial over 
K has only simple roots in K. 

For M £ A4 n (K), the set ofnxn matrices with entries in K, a(M) denotes its 
spectrum, that is the set of its eigenvalues in K. Two matrices A, B £ M n (K) are 
said to be simultaneously triangularizable (denoted by ST) over K if there exists 
a matrix P £ GL n {K) such that P~ 1 AP and P~ 1 BP are upper triangular. Thus 
such matrices have common invariant subspaces that form a complete flag over K. 
Note that if A, B £ M. n (K) commute, then they are ST over an extension field of 
K . In the sequel, L denotes an extension field of K. 

In Section 2, we consider A,B£ Ai n (K) and we assume that A has n distinct 
eigenvalues in L, an extension field of K, and that a (A) is explicitly known. We 
give an algorithm which allows to check whether or not A and B are ST over L. 
Moreover, when A and B are ST we obtain a basis of L that diagonalizes A and 
triangularizes B. 

In Section 3, we assume that A,B£ A4 n (K) have a common invariant proper 
vector subspace of dimension k over L. We recall some criteria for the existence 
of common invariant proper subspaces of matrices. Shemesh gives this efficient 
criterion, when k = 1, in [7] 

Theorem. Let A, B £ A4 n (C). Then A and B have a common eigenvector if and 
only if 



p| ker([^,fl«])^{0}. 
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Note that the complexity of this test is in 0(n 5 ). 
When k > 2, a particular case, that is sufficient for our purpose, is treated in [I], [5] 
as follows. If U G M n (K) 1 then U^ k > denotes its k th compound. 

Theorem 1. Let A,BG Ai n (C), k € [2, n — lj be such that A has distinct eigen- 
values. The following are equivalent 

(i) A, B have a common invariant subspace W of dimension k. 
(ii) There exists s € C such that (A + sl n )^ has distinct eigenvalues and 
(A + sln) 1 --^, (B + sl n )( k ' are invertible and have a common eigenvector in 
eft). Moreover this eigenvector is decomposable in the exterior product of 
k vectors that constitute a basis ofW. 

We can show that the complexity of this test is at most (when k = n/2) in 

( 77,5/2 

Remark 1. The previous two results are also valid over a field that is algebraically 
closed and has characteristic 0. 

In the sequel, we work on a perfect field K and we will use the following notation 

Notation. Let P £ K[x] be an irreducible polynomial of degree n. The splitting 
field Sp of P is K{x\ , ■ ■ ■ ,x n ) where x\ , • • • , x n are the roots of P in K. The Galois 
group of P is the set of the if-automorphisms of Sp, that is 

Gal{S P /K) = {t G Aut(Sp) | V* G K, r(t) = t}; 

it is isomorphic to a subgroup of S n , the group of all the permutations of {1, • • • , n}. 
If M 6 A4 n (K) and xm, the characteristic polynomial of M, is irreducible, then 
Gm denotes the Galois group of xm- 

Assume that A, B £ A4 n (K) have a common invariant proper subspace of di- 
mension k over an extension field L of K and that xa is irreducible over K. We 
consider conditions that imply that A, B commute. We show the following results. 

(i) If k G {1, n — 1}, then A, B commute. 

(ii) If 1 < k < n - 1 and G A = S n or G A = A ni then AB = BA. 

(iii) Ifl<fc<n— 1 and n is a prime number, then AB = BA. 

The idea is as follows : let F = [ui, • • • ,Uk] be a A-invariant vector space where 
the (ui)i<k are eigenvectors of A associated to the eigenvalues E = {cn)^ c a (A). 
We seek elements of Ga so that their orbits contain elements of E and elements of 
<j{A) \ E. We consider Bu\ G F and we show that it is colinear to u\. 
In Section 4, we consider the case when n — 4, k — 2 and we show that the 
conclusion of (ii) may be false if we drop the hypothesis Ga — <?4 or Ga = A4. 
In Section 5, we use (i) and the simultaneous triangularization to solving the matrix 
equation AX — XA = X a in a particular case. 

2. An algorithm checking ST property 

Proposition 1. Let A, B G M n (K) that are ST over L, an extension field of K. 
We assume that A has distinct eigenvalues over L. Then there exists S € GL n (L) 
such that S~ 1 AS is diagonal and S~ 1 BS is upper triangular. 

Proof. There exists P G GL n (L) such that P- l AP = T,p- 1 BP = U where T 
and U are upper triangular. Note that a(A) c L. The principal minors of T are 
diagonalizable over L. By induction, we can construct a T-eigenvectors basis of 
L n such that the associated change of basis matrix is a upper triangular matrix 
Q G GL n (L). Let S = PQ. Then S^AS is diagonal and S^BS = Q^UQ is 
upper triangular. □ 
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Remark 2. We may replace each column of S with a proportional column. 

The previous result leads to an algorithm to check whether two such matrices 
are ST or not. Its complexity is in 0(n 4 ). 

Proposition 2. Let A, B £ M n (K). We assume that A has n distinct eigenvalues 
in L, an extension field of K , and that we know explicitly o~(A). Then we can decide 
if whether or not A and B are ST over L. If A and B are ST over L, then we 
obtain explicitly a matrix S £ GL n (L) that diagonalizes A and triangularizes B. 

Proof. Since A has distinct eigenvalues in L, we can calculate, from o~(A), a A- 
eigenvectors basis of L n . Let R be the associated matrix and Z = Rr l BR — [ztj]. 

Case 1. The matrices A,B are ST. According to the proof of Proposition Q] and 
Remark [2j there exists a permutation matrix D such if S = RD, then 
S~ 1 BS = D~ l ZD is upper triangular. 
We consider the following algorithm 

U:={l,~-,n}. 
For every i < n, 

if the i th column of Z is zero, then on :— n 

else cti := n — sup{j < n | z[j, i] ^ 0}. 
For r from 1 to n, do 

find i r such that = sup^j a, . 

if cti r < n- r then BREAK 

U:=U\{i r }. 
If r = n then OUTPUT := • • • , d n ) 
else OUTPUT := NULL. 

The output (e?i, • • • , d n ) constitutes a convenient permutation. 
Case 2. The matrices A,B are not ST. The previous algorithm gives the output 
NULL. 

□ 

Remark 3. When A has multiple eigenvalues or cr(A) is unknown, to find an 
efficient algorithm is hard. A finite rational algorithm which allows to check whether 
two given n x n complex matrices are ST is exposed in [TJ Theorem 6] . The study 
of complexity of the presented algorithm is omitted in [T] and, as the author shows 
in [2], this test is impractical for n > 6. 

3. Common invariant subspace and commutativity 

Proposition 3. Let A G A4 n (K) such that A has n distinct eigenvalues in an ex- 
tension field L of K and let Z = {B £ A4 n (K) \ A, B have a common eigenvector 
in L n }. Then Z is the union of n subspaces of A4 n (K), each of them containing 
the commutant of A. 

Proof. Let a G cr{A), L a = K[a] and [L a : K\ = k a . Let u E L a n \{0} be such that 
Au = cm. If B — [bij] G Z, then the condition ll Bu and u are linearly dependent" 
can be written in the form of n — 1 L Q -linear conditions on the (pi,j)i,j, that is 
k a x (n — 1) i^-linear conditions on the (bij)ij. Thus B is in a if-vector space of 
dimension at least n 2 — k a (n — 1) that contains the commutant of A. Finally B 
goes through the union of n such subspaces. □ 

Remark 4. One has several interesting properties when \A is irreducible over K 

i) The endomorphism A has no invariant proper subspaces over K . 

ii) Since K is a perfect field, A has simple eigenvalues in S XA and its commutant 
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is K[A] and has dimension n. 

Hi) According to [3 p- 51] and ii), any A-invariant subspace of dimension k over 
K is spanned by k A- eigenvectors. 

From now on, we suppose that A and B have a proper common invariant subspace 
of dimension k over an extension field of K . 

Theorem 2. Let n > 2. Let A, B € M. n {K) be such that they have a common 
eigenvector over an extension field of K . We assume that the characteristic poly- 
nomial of A is irreducible over K. Then AB = BA. 

Proof. Let u be a common eigenvector and put Au — au, Bu = (3u. Recall that 
G a is a transitive group, that is, there exist (Ti)i=i t ... G Ga such that 

a{A) = {a,n(a), ■ ■ ■ ,r n _i(a)}. 

Moreover, Ti(u) is defined componentwise and Afcfo)) — Ti(Au) = Ti(ct)Ti(u). 
Finally {u,T\(u), - • ■ ,t„_i(u)} is an associated basis of eigenvectors of A. Thus 
B{Ti{u)) = Ti(Bu) = n{fi)Ti{u) and AB = BA. □ 

We can slightly improve the previous result as follows. 

Lemma 1. If A, B G A4 n (L) have a common invariant subspace of dimension k 
over L, then A T and B T have a common invariant subspace of dimension n — k 
over L. 

Proof. The common invariant subspace of dimension k, can be written V = {X G 
L n | AX = 0} where A G M n -k,n{L) has maximal rank n — k. Since ker(A) C 
ker(AA), there exists Z G M n -k{L) such that AA = ZA, that is A T A T = A T Z T . 
The n — k columns of A T span a vector space of dimension n — k that is invariant 
for A T . □ 

Corollary 1. Let A, B G M n {K) be such that they have a common invariant hy- 
perplane over an extension field of K. We assume that the characteristic polynomial 
of A is irreducible over K. Then AB — BA. 

Proof. According to Lemma [TJ A T and B T have a common eigenvector and by 
Theorem H A T B T = B T A T , that implies AB = BA. □ 

Now we consider the case where A and B have a common invariant proper 
subspace of dimension > 2. Recall that A n , the group of even permutations of 
{1, • • • , n}, contains the cycles of odd length. 

Theorem 3. Let n > 3 and A, B G A4 n (K) be such that they have a common 
invariant proper vector subspace over an extension field of K . We assume that xa 
is irreducible over K and Ga = S„ or Ga = A n . Then AB = BA. 

Proof. Since \A — Xa t ancl according to Lemma [1] we may change k with n — k 
n 

and assume that k < — , that implies k + 2 < n. Let F be a common invariant 

subspace of dimension k > 2 for A, B. According to Remark[4] Hi), the subspace F 
is generated by certain eigenvectors u\,- ■ ■ ,Uk of A respectively associated to the 
pairwise distinct eigenvalues of A: a%, ■ ■ ■ , a^. Let cr(A) = {ai, • • • , afc, • • • , a n }. 
There exists r G A n C Ga, a cycle of length r = k + 1 if k is even (resp. r = 
k + 2 if k is odd) such that, for every 1 < i < r — 1, = r(ai). Note that 

F = [ui, ■ ■ ■ ,r fe_1 (?ii)] and T k (u\) F. Assume that Bu\ = Yli=o •^ rl ( u i) where 
q G [1, k — lj, for every i, A; G S XA and X q ^ 0. Therefore 

9-1 

Bu k ^ q+1 =B(T k -«( Ul )) =Y J T k - q {K)T k - q+ \u 1 ) + \ q T k {u l ) G F. 

i=0 
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Then X q = 0, that is a contradiction. Finally Bu\ — XqU\ and we conclude by 
Theorem H □ 

We can wonder if we still get the same conclusion of Theorem [3] when droping 
the hypothesis Ga — S n or G a — A n . The answer is no in general but is yes if n 
is a prime. 

Theorem 4. Assume that n > 3 is a prime number and let A, B £ M. n {K) be 
such that xa is irreducible over K. If A and B have a proper common invariant 
subspace, then AB = BA. 

Proof. Let F be a common invariant subspace of dimension fee [2, n — 1] for A, B. 
Let u G F be an eigenvector of A associated to a € p(A). Note that n divides 
the cardinality of Ga- Since n is prime and according to Cauchy's theorem, there 
exist r <G Ga of order n. Necessarily the permutation r is a cycle of length n and 
cr{A) = {a, ■ ■ ■ ,T n ^ 1 (a)}. Moreover {«,••• ,r"~ 1 (u)} is a basis of eigenvectors 
of A and some among these vectors constitute a basis of F. Put Bu = Xqu + 
So<i<n-i AjT l (tt) where the are in K. Assume that there exists p G 1] 
such that X p ^ 0. Since n is prime and k < n, there exists an integer q such that 
r q (u) e F and t«+p(u) g F. Therefore 

B{r q {u)) = T q (X ) T q (u) + T q (Xi)T q+i (u) + r q (X p )T q+ P(u) e F. 

0<i<n,z^p 

Thus B(T q (u)) is written as a linear combination of the basis {u, ■ ■ ■ , r" _1 (u)} and 
the coefficients of the vectors that are not in F are zero. Consequently X p — 0, that 
is a contradiction. Finally Bu = Xou and we conclude by Theorem [2] □ 

Remark 5. Consider A, B € M35(Q) such that AB ^ BA (the verification is easy) 
and Ga — S35 or A35 (the verification is easy with the "Magma" software). Then, 
by Theorem we deduce that A, B admit no common invariant proper subspaces 
(the direct verification is impossible because the algorithm associated to Theorem^ 
is impractical for n > 12). 

4. The case n = 4 

Assume that A, B € M.^{K) have a common invariant subspace of dimension 
k E {1,2,3} and that xa is irreducible over K. If k — 1,3, then from Theorem 

and Corollary [TJ AB = BA. From Theorem we obtain the same conclusion 
if k = 2 and Ga — S4 or A^. It remains to study the cases where A admits an 
invariant plane II and Ga — C4, the cyclic group with four elements, C2 2 or P4, the 
dihedral group with eight elements. Of course, if K is a finite field, then necessarily 
Ga = C4. 

Let A e Ai n (K) be such that xa is irreducible over K and II be a ^-invariant 
plane. We denote by ^(n) the dimension of the if- vector space of the matrices 
B G A4 n (K) such that II is a B-invariant plane. We will see that ^(11) does not 
depend only on k and Ga- 

Proposition 4. Let A € M^K) such that xa is irreducible, Ga = T>4 and II is a 
A-invariant plane. Then rvi(n) =4 or 8. 

Proof. There exist a\,a^ £ &(A) such that LI = ker((A — ail4)(A — 02-^4))- Let 
L = K(ai,ct2). Note that H — {r s Ga \ T(a\) = 1^2} has two elements. Let u 
be an eigenvector of A associated to a\. If t € H, then {u,t(u)} is a basis of II. 
Let B € Al^if) such that II is S-invariant. Therefore -Bit = Au + /ir(w) where 
X,fj, £ L. 
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• Case 1. One element r of H has order 4. Then a (A) = (r I (ai))o<i<3 and 
(r l (w))o<i<3 is a basis of K n constituted by eigenvectors of A. Thus 

B{t{u)) = t{X)t(u) + t{h)t 2 {u) G n, 

that implies /i = 0. Therefore, for every i, B(r t {u)) = t 1 (X)t 1 (u) and 
AB = BA. 

• Case 2. The elements of H have order 2. Then 

B(t(u)) = t{X)t{u) + t(ji)u G II. 

Let (r{)i=3 J 4 G Ga such that Tj(ai) = a*. Clearly, {w, r(u), T3(m), ^(u)} is 
a basis of eigenvectors of A and, for i = 3,4, B(r<(u)) = Ti(Bu) depends 
only on .Bit. Finally £? depends only on A, (i G L and r^II) = 2[L : if]. 
Necessarily 7^(11) < 16 and [L : K] = 4 or 8. Therefore fA(II) = 8. 

□ 

Proposition 5. Let A G •M^if) suc/i £/ia£ \A is irreducible, Ga — C4 and H is a 

A-invariant plane. Then 7^4 (II) =4 or 8. 

Proof. We use the notations of Proposition^ Here [L : K] = 4 and H has a unique 
clement r. 



• Case 1. r is a generator of Ga- As in the proof of Proposition [H Case 1, 
we show that AB = BA. 

• Case 2. r has order 2. As in the proof of Proposition Case 2, we show 
that r A (U) =2[L:K] = 8. 

□ 

Proposition 6. Let A G A^i-Q suc/i i/iai xa is irreducible, Ga = C2 2 and H is 
a A-invariant plane. Then r A (JT) = 8. 

Proof. Again we use the notations of Proposition [H Here [L : K] = 4, H has a 
unique clement r and r has order 2. As in the proof of Proposition |4j Case 2, we 
show that r A (IL) = 2[L : K] = 8. □ 



Example. 

• W^e consider the following instance where K 

l + ev/5 



1 and II e = ker(A 2 



-A- 



P, Xa(^) = x 4 + x 3 + x 2 + 
I4) where e = ±1. Here Ga = C4, the 



A 



where Gb = ©4 



element of H has order 2 and, according to Proposition^ r^(II e ) = 8. In 
particular, the following pair (A, B) is such that the planes IL e are invariant 
for A, B and yet, A and B are not ST. 

(0 -1\ / -10 2^ 

10 0-1 -1-111 

10-1 ' 1 

\p 1 -\j \ 1 00/ 

M^it/i i/ie help of Theorem QJ we show that A, B admits only the planes Tl e 
as proper common invariant subspaces over C. 

i) Applying the Shemesh's criterion to the couples (A,B) and (A T ,B T ), we 
conclude that there are no solutions in dimensions 1 or 3. 

ii) We prove easily that (A + 1^)^ and (B + I^ 2 * 1 have two common 
eigenvectors 

1 -ev/5 + 1 



1. 



-,1] J 



COMMON INVARIANT SUBSPACE AND COMMUTING MATRICES 



7 



An easy calculation shows that u e is the exterior product of the vectors of 
a basis of II e . 

• Now we assume K = Z/7Z and \A is as above. Then xa is irreducible and 
since K is finite, G a = C4. Moreover 5 is not a square in K and we can 
define y/E in an extension field of K of dimension 2. Then the planes II e; as 
above, are A-invariant. By the previous reasoning, we obtain r,i(II e ) = 8. 
The matrices A, B , as above, admit the planes IT as common invariant 
subspaces. We can show that A, B have no other proper common invariant 
subspaces. Note that Xb{%) — {x 2 — x + 4)(x 2 + 2x + 2) and B admits two 
invariant planes over K . 

5. Solving a matrix equation 

We give an application of Section 3 using the following known result 

Theorem. (McCoy's theorem [6],) Let L be an algebraically closed field and A, B € 
A4 n (L). Then A,B are ST over L if and only if for any polynomial p(X, fi) in 
non-commuting indeterminates, p(A, B)(AB — BA) is nilpotent. 

Proposition 7. Let A = (J 7 ° p ; 9 ) G M p+q {K) be such that (U, V) G M p (K)x 

M. q (K), xu and Xv are distinct irreducible polynomials over K . If B G A4 p + q (K), 
then A, B are ST over K if and only if B is in the form 

B = Q ) respectively B = °»« 

where Q € A4 Ptq (K), respectively Q £ M q ^ p (K) and K[x] are arbitrary. 

Proof. Clearly cr(A) — {a(U),a(V)} and A has p + q distinct eigenvalues. 

The eigenvectors of A are in the form [u, 0] T where u T is an eigenvector of U or 
[0, v] T where v T is an eigenvector of V. Note that A, B have a common eigenvector 
and assume, for instance, that it is in the form [u, 0] T with Uu T = au T . We 
adapt the proof of Theorem [2j there exist (Tj)j = i... p G Gjj such that a(U) — 
{a,Ti(a), ■ ■ ■ ,r p _i(a)}. We deduce that the ([rj(u), 0] T )i< p are eigenvectors of B 

and B is in the form B = ( P ® ) where UP = PU, that is P = f(U) E K[U). 



Then AB — BA = ( 5* p m , ] and, more generally 

Vq,P y ti— HV 



p(A, B)(AB - BA) 



P * 



v 9 , p p{V,R){VR-RV) / 

where p(X, fi) is any polynomial in non-commuting indeterminates A, fi. According 
to the McCoy's theorem, A, B are ST over K if and only for any polynomial p(X, fi) 
in non-commuting indeterminates, p(V, R)(VR—RV) is nilpotent, that is equivalent 
to : V, R are ST. Then V, R have a common eigenvector and, according to Theorem 
[2j VR — RV, that is R is a polynomial in V. 

(<=) Again using the McCoy's theorem, the converse is clear. □ 

Finally we apply Proposition [7] to solving a matrix equation. 

Proposition 8. Letp,q be distinct positive integers. Let A £ M. p + q (K) be such 
that XA = < £"i' where $ and ^Sf are polynomials of degree p and q, irreducible over 
K . Let a be a positive integer. Then the equation, in the unknown X G Ai p + q (K), 

(1) AX — XA = X a 

admits the unique solution X = 0. 
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Proof. We may assume that A — ( „ ^ 9 J where U, V are the companion 

yjq,p V J 

matrices of $, Since X satisfies Equation (|T|), A and X are ST over K (cf. 

According to Proposition [71 necessarily X has two possible forms, for instance this 

one 

X = ({ (U) ^ and consequently AX - XA = ( °> U Q QV 

\%, q g(v)J H \p q , P % 

i) Assume a = 1. Equation ^ reduces to 

f(U) = 0, g(V) = 0, UQ-QV = Q. 

The last equation can be rewritten <fi(Q) = Q where = U ® I q — I p ® F T is the 
sum of two linear functions that commute. Therefore 

<t(0) = {A- M | Aea(C7) )M e<T(F)}. 

If there are non-zero solutions, then there exist A G cr([/),/i g cr(F) such that 
A — /i = 1. Since is the minimum polynomial of A over K, then xu{x + 1) is the 
minimum polynomial of fi over K and X(7( a; + 1) — Xv{x). That implies p = q, a 
contradiction. 

u) Assume a > 1. Equation ((TJ reduces to 

f(U) a = , g(V) a = 0,UQ-QV = 0. 

Let <r([/) = (Ai)i<p. Then (f(\i))i< p = <r(f(U)) = {0}. Since / is a unitary 
polynomial of degree p, /(x) = (a; — Ai) • • • (x — X p ). By Cayley-Hamilton Theorem, 
f(U) = and, in the same way, g(V) = 0. By the reasonning used in i), for every 
A e a(A),fi e c(B), A — 7^ and is a linear bijection. We conclude that 
Q = 0. □ 
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